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Today's world is no longer that of yesterday, the pace with which we live 
and also the speed is enormous and rapid, that overnight we discover the 
appearance of new technologies and solutions in all the fields, in particular, 
that of scientific research. Artificial intelligence plays the main role. 
Predicting the behavior of new materials using artificial neural networks 
(ANNs) has become a frequently adopted solution by researchers today. The 
performance of neural networks depends mainly on the activation functions 
used. This work was designed to mainly study the impact of these functions 
on the response speed of an ANN in general, and particularly on the model 
we are working on to predict the thermomechanical behavior of innovative 
materials. By using tansig, purelin and logsig in a feed forward back 
propagation training by Levenberg-Marquardt algorithm, we were able to 
generate 9 models. For each of these models, we were interested in 
analyzing the speed’s response of the network and studying its regression. 
Thus, this work was able to show us that choosing the right neuron 
activation function from one layer to another can clearly influence the 
performance of the results. Depending on the problem studied, the desired 
objective and the chosen architecture, the activation function can radically 
change the result and provide us with the expected efficiency. 
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1. INTRODUCTION 


Materials science has always been one of the flagship strands of scientific research because the use 
of matter is found in all areas of human life. It was able to follow and take advantage of the solutions offered 
by technology through the different eras. At the beginning the goal of the researchers was to discover the 
elements and materials existing on earth, to study them, to analyze them and to optimize their applications. 
More recently, their objective is to go beyond the existing and innovate new non-existent materials [1], 
model them, study them and analyze them to use them for very specific applications; robotics, aviation, 
automotive, food and others. The combination of two materials whose properties we already know can give 
us a new one that is more efficient than the other two in terms of characteristics. 

At the same time, artificial intelligence (AI) that was designed to mimic human intelligence and go 
beyond human ability, reasoning and logic represents the future in every way. AI can be used in 
classification, optimization also in prediction and this in different fields; in medicine for example to predict 
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breast cancer [2], in the field of quality to predict the results of an audit control [3] or in materials science, to 
predict the properties of innovative materials, bio composites, in our case. AI has already shown its ability to 
give satisfactory results in predicting the elasticity of a biocomposite to a certain degree of percentage of the 
mixture and this using artificial neural networks [4], [5]. The performance of an artificial neuron network 
(ANN) is linked to several parameters, the architecture of the model chooses, number of neurons in each 
layer, the weight of the bonds, but particularly it is linked the activation function of neurons. An activation 
function is defined as a mathematical function allowing the passage of information from one neuron to 
another [6]. In the same neural network, we can find more than an activation function because the latter can 
be different from one layer to another. 

This work aims to study the response of the neural network of choice using three types of activation 
functions popularly used in AI, namely tansig, logsig and purelin. By using a single architecture with these 
three activation functions, 9 models will be generated and studied to analyze them in order to allow us to 
converge towards the fastest and most precise result, and therefore lead us to the most optimum model to 
predict the behavior of innovative materials. 


2. Dataset and model building 
2.1. Dataset 

AI relies on the results of the latest research using different analytical techniques to arrive at the 
desired results. So, to train an ANN you need to have a dataset with the data as input and the desired goal as 
output as information. The objective of training a neural network is to find the relationship between the input 
and the target provided in the dataset, so that when asked to predict a result, it will be able to provide us with 
the desired information with a negligible degree of error [7]. Our dataset represents the results from the use 
of a semi-analytical method called Mori Tanaka [8]. It is made up of two inputs representing the elasticity of 
the two components of the new material, and the output representing its elasticity. 


2.2. Architecture and training function 

In this step we have to choose the architecture of the neural network to build the 9 models to study. 
This process consists in defining the topology of the network, therefore the choice of the number of hidden 
layers and the number of neurons in each layer. Then choose the algorithm to use to train the network. 

In our study, the choice fell on a network of three layers as shown in Figure 1. An input layer, an 
output layer and a hidden layer. The number of neurons in the input layer is identical to the number of input 
variables of our dataset which is 2 as well as the number of neurons in the output layer is 1 representing the 
target of our datasast. Regarding the hidden layer, our choice fell on 15 neurons. This choice was not 
arbitrary, it was after several trials and studies that we made this choice [9]. So, our network topology is 2- 
15-1. Since this is a regression problem, we opted for the feed forward back propagation trained with 
Levenberg-Marquardt (ML) algorithm since the latter was able to prove its efficiency by combining it with 
the feed forward back propagation [10]. 


Hidden Layer 


15 1 


Figure 1. The network architecture to be studied 


2.3. Activation functions 

At the same time, the activation functions (AF) used in the hidden layer and in the output, layer is 
tansig, logsig and purelin [11]. These three functions allowed us to build 9 models to study. In every three 
models, we set the activation function of the hidden layer and we modify that of the output layer, and so on 
for the other two functions used. In Table | we find the representation of the models with the activation 
functions used in the two hidden layers and the output one. 
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Table 1. Different activation functions used on the proposel models 
Model number Activation functions hidden layer/output layer 
tansig/logsig 
tansig/purelin 
tansig/tansig 
logsig/tansig 
logsig/tansig 
logsig/purelin 
purelin/tansig 
purelin/logsig 
purelin/purelin 


COAAINDNKRWNE 


—  Tansig: hyperbolic tangent sigmoid transfer function: characterized by its wider interval which can 
ensure rapid learning. It is defined as, 


f(x)=Tanh (x) = {= e 1} (1) 


1+e72* 


—  Logsig: log sigmoid transfer function. It is defined as, 


1 


f(x)=logsig (x) = [i= (2) 
—  Purelin: linear transfer function is typically used for function approximation and regression tasks. It is 
defined as, 
f(x)=Purelin (x) = x (3) 


3. RESULT AND DISSCUTION 
3.1. Intelligent model’s number 1, 2 and 3 

This is a 2-15-1 topology feed forward back propagation trained with a ML. In these three models 
we used and fixed tansig as the activation function of the hidden layer [12], and we changed that of the 
output layer in each model. Table 2 shows us the responses of the network studied in relation to these 
activation functions. The training of the latter has been done in such a way that at each training epoch we 
save the trained model by performing a check to avoid overfitting [13]. Among these three models, the most 
suitable combination giving a faster result is that of the third model, followed by the combination of tansig 
with tansig. 


Table 2. The response time of the three models using the AF tansig in the hidden layer 
Model number AF hidden layer/ AF output layer Average time (s) Min time (s) Max time (s) 


1 tansig/logsig 4.32 0.82 7.83 
2 tansig/purelin 1.03 0.60 1.47 
3 tansig/tansig 0.91 0.52 1.30 


A representation of the evolution of the regression coefficient over time for the three models in 
question clearly shows us that the fastest model is also the one which converges quickly to | [14]. The 
regression coefficient is one of the important indicators proving the performance of an artificial neuron 
network [15], the closer this coefficient is to 1, the more the network is said to be efficient. Figure 2 
illustrates the result clearly. And from Figure 2, we can deduce that when we are looking for regression 
coefficients close to 1 exhibiting more efficiency, the Tansig/Tansig function quickly leads to this result. 


3.2. Intelligent model’s number 4, 5 and 6 

This is a 2-15-1 topology feed forward back propagation trained with an ML. In these three models 
we used and fixed logsig as the activation function of the hidden layer [16], and we changed that of the 
output layer in each model. Table 3 shows us the responses of the network studied in relation to these 
activation functions. The training of the latter has been done in such a way that at each training epoch we 
save the trained model by performing a check to avoid overfitting. Among these three models, the most 
suitable combination giving a faster result is that of the sixth model. 
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Figure 2. The convergence of the regression coefficient in function of time 


Table 3. The response time of the three models using the AF logsig in the hidden layer 
Model number AF Hidden Layer/ AF Output layer Average time (s) Min time (s) Max time (s) 


4 logsig/tansig 12.81 3.12 22.5 
5 logsig/logsig 19.78 6.9 32.66 
6 logsig/purelin 7.24 4.36 10.13 


Like wise for these three models using logsig in the hidden layer, it was necessary to follow the 
evolution of the regression coefficient of each model as a function of time. The representation of the latter in 
Figure 3 shows us that the three models do not converge. This is due to the use of the logarithmic function in 
the hidden layer. This activation function was able to provide us with results, but it showed us that its use in 
the hidden layer will not be able to provide us with effective results. 
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Figure 3. The convergence of the regression coefficient in function of time 


3.3. Intelligent model’s number 7, 8 and 9 

This is a 2-15-1 topology feed forward back propagation trained with an ML. In these three models 
we used and fixed purelin [17] as the activation function of the hidden layer, and we changed that of the 
output layer in each model. Table 4 shows us the responses of the network studied in relation to these 
activation functions. The training of the latter has been done in such a way that at each training epoch we 


save the trained model by performing a check to avoid overfitting. Among these three models, the most 
suitable combination giving a faster result is that of the ninth model. 


Table 4. The response time of the three models using the AF purelin in the hidden layer 
Model number AF hidden layer/ AF output layer Average time (s) Min time (s) Max time (s) 


7 purelin/tansig 0.82 0.52 1.12 
8 purelin/logsig 1.9 1.7 2.1 
9 purelin/purelin 0.49 0.13 0.85 
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3.4. Fastest model validation 

In this part, we will proceed to the validation of the fastest model, and this by asking it to predict the 
behavior of a new material. Polypropylene reinforced with 15% biofiller [18]. The ANN was trained by 
testing and error checking in such a way as to see the minimum squared error along with the maximum 
regression value. Using 1,000 repeated epochs of training, the regression produced by our model is shown in the 
Figure 5. The regression coefficient of polypropylene [19] reinforced at 15% with a choosen bio load is 0.99 


and that converges to 1 and signified that our model is efficient and performant [20]-[25]. 


Output ~= 1*Target + -65 


Output ~= 1*Target + 3.2 


4. CONCLUSION 

In this article we worked on an artificial neural network that meets the need to predict the properties 
of a new material, to study its response by playing on the activation functions of neurons. The activation 
function of a neuron is a very important parameter that influences the results. Choosing the right activation 
function clearly leads to effective results. The aim was to study the behavior of a feed forward back 
propagation which consists of three layers. Using tansig, purelin and logsig in activating neurons from both 
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Figure 5. Performance validation by regression analysis 
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Figure 4. The convergence of the regression coefficient in function of time 
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hidden and output layers, we were able to build 9 models that we studied. The speed of response as well as 
the regression of each of this model was our goal. We were able to show that in the prediction of the 
characteristics of biocomposites, the use of the tansig/tansig function allowed us to have a good regression 
coefficient close to 1 more quickly than using the other combinations of the functions of activation. The 
activation function remains a mechanism by which neurons process and transmit information through the 
different layers. For the tansig, the processing and the transmission are done in a wide interval which allows a 
fast learning to converge towards a better regression. 
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